Conversation
Cattrs works by passing the value it is structuring into the default
constructor of the target type, so it supports Python's coercions.
In this case, consider the previous version of TextData:
class TextData:
data: str
type: "text" | "file"
When a test suite containing {'type': 'text', 'data': 500} is
structured, cattrs will attempt to structute `data` by doing `str(500)`.
This is legal, so cattrs happily accepts it.
The new definition, however, is as follows:
class TextData:
content: ContentPath | str
path: str | None = None
When the decorator runs and returns the value from data as an int,
cattrs will take over, trying to structure it into `ContentPath | str`.
TESTed has a custom union structuring function to allow structuring
into any union. One thing it does not do, however, is the data coercion.
The union structuring requires exact types, which is why the code failed
and the old test suite no longer works.
To fix this, we now explicitly coerce the legacy data to a string inside
the fallback converter before it reaches the strict union hook.
Another attempt for `stdin`
Support `input_files`
Add support for `output_files`
There was a problem hiding this comment.
Pull request overview
This PR introduces end-to-end “file features” across the TESTed ecosystem (schema/DSL parsing, testsuite model, planning/execution, and oracle evaluation), enabling input files, stdin-as-file, and multi-file output comparisons.
Changes:
- Refactors suite data structures to model text/file content via
TextData(content, path)and introduces multi-fileFileOutputChannel(files=[...]). - Extends DSL + JSON schema to support
stdinas file data,input_files,output_files, and!pathreferences. - Updates planning/execution/evaluation/oracles to create/copy dynamic files and to report per-file oracle results.
Reviewed changes
Copilot reviewed 70 out of 70 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/tested-draft7.json | Extends schema test types with path. |
| tests/test_testsuite_legacy.py | Adds legacy-format parsing regression tests. |
| tests/test_suite.py | Updates tests to content/files[] model. |
| tests/test_planning_sorting.py | Adds tests for dynamic-file collection/sorting/dedup. |
| tests/test_planning.py | Adds planning tests for file/stdin/output conflicts and splits. |
| tests/test_parsing.py | Adds tests for new parsing decorators and converter hooks. |
| tests/test_oracles_programmed.py | Adds programmed-oracle coverage for output files. |
| tests/test_oracles_builtin.py | Updates built-in oracle tests for new file oracle + multi-file. |
| tests/test_io_exercises.py | Adds IO exercise plans covering stdin + dynamic files. |
| tests/test_functionality.py | Updates stdin tests to TextData(content=...). |
| tests/test_file_linking.py | Adds tests for file-link messages and readable input linking. |
| tests/test_dsl_yaml.py | Updates DSL YAML tests and adds stdin/input_files cases. |
| tests/test_dsl_legacy.py | Adds DSL legacy compatibility coverage for files/output files. |
| tests/test_collector.py | Ensures per-file NOT_EXECUTED reporting on termination. |
| tests/exercises/time-2-code/solution/solution.ts | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/solution/solution.sh | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/solution/solution.py | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/solution/solution.kt | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/solution/solution.js | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/solution/solution.java | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/solution/solution.hs | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/solution/solution.cs | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/solution/solution.cpp | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/solution/solution.c | New fixture solution for file I/O exercise. |
| tests/exercises/time-2-code/evaluation/plan.yml | New evaluation plan using input/output files. |
| tests/exercises/output-files-custom-oracle/solution/wrong.ts | New wrong-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/wrong.py | New wrong-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/wrong.kt | New wrong-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/wrong.js | New wrong-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/wrong.java | New wrong-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/wrong.cs | New wrong-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/wrong.cpp | New wrong-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/correct.ts | New correct-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/correct.py | New correct-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/correct.kt | New correct-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/correct.js | New correct-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/correct.java | New correct-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/correct.cs | New correct-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/solution/correct.cpp | New correct-solution fixture for custom file oracle. |
| tests/exercises/output-files-custom-oracle/evaluation/suite.yaml | New suite using output_files + custom oracle. |
| tests/exercises/output-files-custom-oracle/evaluation/evaluator.py | Custom oracle implementation fixture. |
| tests/exercises/echo/solution/correct-files.ts | Adds file-aware echo solution fixture. |
| tests/exercises/echo/solution/correct-files.sh | Adds file-aware echo solution fixture. |
| tests/exercises/echo/solution/correct-files.py | Adds file-aware echo solution fixture. |
| tests/exercises/echo/solution/correct-files.kt | Adds file-aware echo solution fixture. |
| tests/exercises/echo/solution/correct-files.js | Adds file-aware echo solution fixture. |
| tests/exercises/echo/solution/correct-files.java | Adds file-aware echo solution fixture. |
| tests/exercises/echo/solution/correct-files.hs | Adds file-aware echo solution fixture. |
| tests/exercises/echo/solution/correct-files.cs | Adds file-aware echo solution fixture. |
| tests/exercises/echo/solution/correct-files.cpp | Adds file-aware echo solution fixture. |
| tests/exercises/echo/solution/correct-files.c | Adds file-aware echo solution fixture. |
| tests/exercises/echo/evaluation/plan.yaml | Adds evaluation plan covering stdin-as-file. |
| tests/exercises/echo/evaluation/plan-dynamic.yaml | Adds evaluation plan covering dynamic files. |
| tests/exercises/echo/evaluation/input.txt | Adds resource fixture for !path stdin. |
| tested/testsuite.py | Core suite model refactor for TextData and multi-file output. |
| tested/parsing.py | New hook-chaining mechanism and updated fallback/ignore decorators. |
| tested/oracles/value.py | Extends value extraction to support file output channels. |
| tested/oracles/text.py | Extracts file oracle out; exposes text_options. |
| tested/oracles/file.py | New multi-file-aware file oracle implementation. |
| tested/oracles/common.py | Supports multi-result oracles + channel override reporting. |
| tested/oracles/init.py | Routes file builtin oracle to new oracles.file. |
| tested/languages/preparation.py | Updates before/after handling to TextData(content=...). |
| tested/languages/generation.py | Updates readable input + linking for input_files/stdin-path. |
| tested/judge/planning.py | Tracks input/output file conflicts; collects dynamic files per unit. |
| tested/judge/execution.py | Materializes dynamic files into execution directories. |
| tested/judge/evaluation.py | Supports per-file oracle results and new file-link messaging. |
| tested/dsl/translate_parser.py | Adds !path, stdin file objects, input_files/output_files. |
| tested/dsl/schema.json | Updates DSL schema for file data and output files. |
| tested/dsl/schema-strict.json | Updates strict DSL schema for file data and output files. |
| tested/configs.py | Updates @fallback_field decorator usage to new API. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
I've generated test suites for all Time2Code's level 08 exercises. Those exercises make optional use of the file extensions that will be added to TESTed in this PR. Some test cases i) use multiple input/output files and ii) use the same file name with different content. The last feature is not supported by the original Python judge, so testing these exercises wasn't possible with the original judge. I'll test these level 08 exercises as soon as this PR has been put in production, and then work out the exercises from level 09 too. |
Revert "Merge pull request #619 from dodona-edu/files-features"
Branch with all file features, so we can merge in one go (and revert in one go if needed)